On the Non-Generalizability in Bug Prediction

Bug prediction is a technique used to estimate the most bug-prone entities in software systems. Bug prediction approaches vary in many design options, such as dependent variables, independent variables, and machine learning models. Choosing the right combination of design options to build an e↵ective bug predictor is hard. Previous studies do not consider this complexity and draw conclusions based on fewer-than-necessary experiments. We argue that each software project is unique from the perspective of its development process. Consequently, metrics and machine learning models perform di↵erently on di↵erent projects, in the context of bug prediction. We confirm our hypothesis empirically by running di↵erent bug predictors on di↵erent systems. We show there are no universal bug prediction configurations that work on all projects.